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OVERVIEW 



THE SENIOR URBAN EDUCATION 
RESEARCH FELLOWSHIP PROGRAM 

Large urban public school districts play a significant 
role in the American education system. In 2007-08, 
the largest 65 urban school systems in the country 
- comprising less than one half of one percent of the 
nearly seventeen thousand school districts that exist 
across the United States - educated about 1 4 percent of 
the nation's K-12 public school students, approximately 
one third of its African American students, a third of 
its English Language Learners, a quarter of its poor 
students, and a quarter of its Hispanic students. They 
also employed nearly 1 5 percent of the nation's teaching 
force. Clearly, any attempt to improve achievement and 
to reduce racial and economic achievement gaps across 
the United States must involve these school districts as a 
major focus of action. 

These school districts face a number of serious, 
systematic challenges. However, solutions to these 
problems are not always obvious, and the existing 
research base is not always sufficient to address them. 
In order to better understand the problems in urban 
education and to develop more effective and sustainable 
solutions, urban districts need a regular program of 
rigorous scientific inquiry focusing on what works to 
improve academic outcomes in the urban context. 
Moreover, in order to produce such evidence and to move 
public education forward generally, the standards of 
evidence in education research must be raised in such a 
way as to bring questions regarding the effectiveness of 
educational interventions and strategies to the fore and 
to promote careful scrutiny and rigorous analysis of the 
causal inferences surrounding attempts to answer them. 

It has been argued that, in order to move such an effort 
forward, a community of researchers, committed to a 
set of principles regarding evidentiary standards, must 
be developed and nurtured. We contend further that, in 
order to produce a base of scientific knowledge that is 
both rigorously derived and directly relevant to improving 
achievement in urban school districts, this community of 
inquiry must be expanded to include both scholars and 
practitioners in urban education. 



Though a great deal of education research is produced 
every year, there is a genuine dearth of knowledge 
regarding how to address some of the fundamental 
challenges urban school districts face in educating 
children— working to close achievement gaps and 
striving to meet the challenges of No Child Left Behind. 
Moreover, while there is a history of “process-related” 
research around issues affecting urban schools, relatively 
few studies carefully identify key program components, 
document implementation efforts, and carefully examine 
the effects of well designed interventions in important 
programmatic areas on key student outcomes such as 
academic achievement. In sum, there is an absence of 
methodologically sound, policy relevant research to help 
guide practice by identifying the conditions, resources, 
and necessary steps for effectively mounting initiatives 
to raise student achievement. 

In order to address this need, the Council of the Great City 
Schools, through a grant from the Institute of Education 
Sciences, established the Senior Urban Education 
Research Eellowship (SUERE) program. 

The Senior Urban Education Research Eellowship was 
designed to facilitate partnerships between scholars and 
practitioners focused on producing research that is both 
rigorous in nature and relevant to the specific challenges 
facing large urban school districts. We believe such 
partnerships have the potential to produce better, more 
practically useful research in at least three ways. First, 
by deepening researchers’ understanding of the contexts 
within which they are working, the program may help them 
maximize the impact of their work in the places where it is 
needed the most. Second, by helping senior staff in urban 
districts become better consumers of research, we hope 
to increase the extent to which the available evidence 
is used to inform policy and practice, and the extent to 
which urban districts continue to invest in research. Third, 
by executing well designed studies aimed at the key 
challenges identified by the districts themselves, we hope 
to produce reliable evidence and practical guidance that 
can help improve student achievement across all urban 
school districts nationally. 
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The primary goals for the Senior Urban 
Education Research Fellowship are to; 

• promote high quality scientific inquiry into the 
questions and challenges facing urban school 
districts; 

• facilitate and encourage collaboration, 
communication, and ongoing partnerships 
between senior researchers and leaders in urban 
school districts; 

• demonstrate how collaboration between scholars 
and urban districts can generate reliable results 
and enrich both research and practice; 

• produce a set of high quality studies that yield 
practical guidance for urban school districts; 

• contribute to an ongoing discussion regarding 
research priorities in urban education; and 

• promote the development of a “community of 
inquiry”, including researchers and practitioners 
alike, committed to both a set of norms and 
principles regarding standards of evidence and a 
set of priorities for relevant, applied research in 
urban education. 

The following volume of the Senior Urban Education 
Research Fellowship Series documents the work of Dr. 
Becky Smerdon, working in conjunction with the DC 
Public Schools (DCPS). Both the research and reporting 
is the sole intellectual property of Dr. Smerdon, and 
reflects her personal experience and perspective as 
an education researcher working in collaboration with 
DCPS. 



Dr. Smerdon encountered some of the greatest 
challenges to working with large urban school district 
data and staff— a lack of thorough, reliable extant student 
data, staff turnover, and a number of high-profile reform 
initiatives each competing for resources and attention. 
These challenges are by no means unique to DCPS. 
Moreover, we feel Dr. Smerdon demonstrated one of 
the most important elements in collaborative education 
research: flexibility. As she points out in the next section, 
her implicit goal in creating an early warning indicator 
system was to support the district's improvement efforts. 
DCPS is currently on the right track in creating reliable, 
valid student datasets, and to becoming a “data-driven” 
district. Certainly Dr. Smerdon observed and documented 
promising practices in data use being pursued by 
individual schools. But, the district-wide data necessary 
for executing the project Dr. Smerdon originally designed 
were not available as we initially thought they would 
be. So having reached a methodological “dead end” 
with the development of an early warning indicator, she 
went about producing a detailed audit and report on 
current data collection and use that could assist the 
district in understanding their current data limitations, 
as well as a collection of practical lessons and policy 
recommendations for establishing a foundation for data 
use in schools. 

With so many districts at precisely this same, preliminary 
stage of building a data culture within their schools and 
at the central office level, we feel this work is particularly 
timely. We hope you will find it interesting and relevant to 
your own work. Thank you. 

Michael Casserly 

Executive Director 

Council of the Great City Schools 
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ABOUT THE SENIOR URBAN 
EDUCATION RESEARCH FELLOW 



Becky Smerdon is Managing Director of Education Research and Policy at 
Quill Research Associates, LLC. She was previously a Principal Research 
Scientist, Vice President and Deputy Director of U.S. Education and 
Workforce Development at the Academy for Educational Development, 
where she was leading the development of a research and development 
agenda on disadvantaged youth and education reform with a particular 
focus on successful transition to college and work. 

Prior to working for the Academy for Educational Development, Smerdon was a Senior Research Associate at the Urban 
Institute where she led a formative and summative evaluation of Baltimore's high school reform initiative, a study of the 
math/science pipeline in North Carolina's reforming high schools funded by the National Science Foundation, and a 
project developing indicators of high school reform implementation funded by the Bill & Melinda Gates Foundation. Before 
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led the U.S. Department of Education's National High School Center. She is a nationally recognized expert in high school 
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College Record, and Research in the Sociology of Education and Socialization. 
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ABOUT THE RESEARCH PARTNERSHIP 



In 2008, we began a project designed to identify the roots 
of the dropout problem in DC Public Schools (DCPS) by 
identifying middle grades students’ exhibiting behaviors 
associated with dropping out of high school. Because 
the drop out problem is most pronounced in ninth grade, 
we chose to focus on the years just prior to ninth grade- 
middle school— and the transition from middle to high 
school. Specifically, our plan was to use DC Public Schools’ 
extant data to create indicators of high school readiness 
by (1) identifying successful and unsuccessful high school 
students and then (2) using their middle school records to 
identify the middle school-level academic characteristics that 
distinguish successful high school students from their less 
successful counterparts. We planned to derive an indicator 
from these middle school characteristics that would serve as 
an “early warning signal” to distinguish between DC middle 
school students who are and are not ready tor high school 
and more pointedly, those who are at risk of dropping out of 
high school. 

Ourfirst step in this project was to conduct athorough inventory 
of the DCPS data, provided by the Center for Education 
Policy Research at Harvard University. Supplemented with 
data from the Common Core Data (CCD) and the DC Public 
Charter Board, we examined student- level data from grades 
6-9 between fall 2005 and spring 2008. We found that 
the dataset had little missing information. Furthermore, we 
were able to supplement much of the data that were missing 
with information from the CCD, Census, and other external 
sources. Other than a few variables, we found the dataset 
aligned well with data found in other datasets, including the 
CCD. However, we concluded that a tew variables should be 
used with caution, if at all— the special education indicator, 
attendance, mobility, and withdrawal information. For 
example, the special education data in 2007-08 indicated 
that all students qualified for services.' While this may have 
been a result of an error in the way the data were pulled from 
the system rather than a representation of the completeness 
of the data in the system, it presented a serious obstacle to 
our analysis. Moreover, withdrawal data were missing a key 
indicator needed to calculate withdrawal rates in 2007-08.^ 
Finally, attendance data— a major component in early warning 
indicators developed in other jurisdictions— indicated that 
there were no absences in 2006-07; absenteeism data in 
other years may be underestimated as they appear lower 



than expected compared with other jurisdictions such as 
Baltimore City Public Schools and Prince Georges County. 
Equally troubling was that only 23 percent of the students 
in the dataset had three years ot consecutive data,^''' which 
is not an adequate representation of the students in the 
District. Thus, we concluded that the DC Public Schools’ 
extant data are not, at this time, suitable tor the early warning 
indicator we had proposed to develop. 

We expect, however, that the data necessary tor the project 
we designed will be available in the future; Chancellor 
Rhee, as well as DCs Office of the State Superintendent 
of Education, has been working to create reliable, valid, 
longitudinal datasets. And while our initial and explicit goal 
of this project was to use data to investigate the location, 
degree, and sources of the dropout problem in DC public 
schools using an early warning indicator system, our implicit 
goal has always been to support and enhance the current 
and future school improvement efforts of DCPS, including 
schools’ use of data. To this end, we revised our project goals 
to examine some of the data problems we uncovered when 
we conducted our inventory. Specifically we visited a sample 
of K-8 schools in the District to learn more about how they 
collect and report data, as well as how they use data to guide 
their own decision making. 

DCPS Chancellor Michelle Rhee supported our efforts, 
signing a letter of commitment on behalf of the district to 
participate in the project. Over the past two years, we met 
with staff from DCPS’ Office of Data and Accountability 
on numerous occasions to describe the purpose of the 
project, request data, share preliminary and final results, and 
seek guidance on findings and their meanings. Elizabeth 
Cohen (formerly of DCPS, now working at DC’s Office ot 
State Superintendent of Education) and Hella Bel Hadj 
Amor (DCPS) were especially generous with their time. On 
occasion, we also met with staff in other offices (e.g.. Office 
of Chief Academic Officer) by invitation of Ms. Cohen and 
Ms. Bel Hadj Amor. All draft and final reports and briefs 
were shared with Ms. Cohen or Ms. Bel Hadj Amor prior 
to submission to the Council of Great City Schools. And 
although DCPS has been undertaking one of the most 
ambitious and controversial reform initiatives in the country, 
support for this project never wavered and staff never 
hesitated to make time to talk. 



1 To correct the issue of special education status, students were coded based on prior year's data, if available. Other special education variables were also used: Special Education Status and 
Special Education Type to verify or correct the designation status. Status and Type were usually only utilized if a student qualified for services. So, if Type or Status was used, the student 
was coded as qualifying for special educational services, but if Type or Status were blank, the students were coded as not qualifying. If Status indicated that a student had exited the special 
education program, the student was coded as not qualifying. This correction yielded a percentage not considerably lower than previous years and one that was comparable to CCD. 

2 Withdrawal rates were calculated In 2005-06 and 2006-07 using Withdrawal Date and Reason for Withdrawal. In 2007-08, there was no date for Withdrawal Date so if a reason was given, 
the student was counted as a withdrawal. This most likely yielded a conservative percentage of withdrawals. 

3 Across the dataset 45 percent of students attending public schools had only one year of data. While 54 percent had at least two years, these data may or may not be consecutive. 

4 Please note these percentages only Include those students that attended public schools for more than one year and do not contain students that attended a public school one year, a charter the 
next and an alternative school the following year (or any other plausible combination). Public schools were the focus because of the completeness of their data in the dataset 
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EXECUTIVE SUMMARY 



With a grant from the Council of the Great City Schools' 
Senior Urban Education Research Fellowship Program, 
we began a project designed to identify the roots of the 
dropout problem in the District of Columbia by identifying 
middle grades students' exhibiting behaviors associated 
with dropping out of high school. Our plan was to use 
DC Public Schools' extant data to create indicators of 
high school readiness by (1) identifying successful and 
unsuccessful high school students and then (2) using 
their middle school records to identify the middle school- 
level academic characteristics that distinguish successful 
high school students from their less successful 
counterparts. We planned to derive an indicator from 
these middle school characteristics that would serve 
as an “early warning signal" to distinguish between DC 
middle school students who are and are not ready for 
high school and more pointedly, those who are at risk of 
dropping out of high school. 

After meetings with DCs Office of Assessment and 
Accountability, we conducted a thorough inventory of 
the DC Public Schools data, provided by the Center 
for Education Policy Research at Harvard University. 
Supplemented with data from the Common Core Data 
(CCD) and the DC Public Charter Board, we examined 
student- level data from grades 6-9 between fall 2005 
and spring 2008. We concluded that the DC Public 
Schools' extant data are not, at this time, suitable for the 
early warning indicator we had proposed to develop. For 
example, attendance data— a major component in early 
warning indicators developed in other jurisdictions— 
indicated that there were no absences in 2006-07; 
absenteeism data in other years may be underestimated 
as they appear lower than expected compared with other 
jurisdictions such as Baltimore City Public Schools and 
Prince Georges County. Equally troubling was that only 
23 percent of the students in the dataset had three years 
of consecutive data,^'® which we did not believe to be an 
adequate representation of the students in the District. 

Through our communications with the district and a 
review of district strategies, we also concluded that the 
Chancellor and her staff have been working diligently to 



create reliable, valid, longitudinal datasets. In an effort 
to support their work, we revised our project goals to 
examine some of the data problems we uncovered when 
we conducted our inventory. Specifically, we visited a 
sample of K-8 schools in the District in the spring of 
2009 to learn more about how they collect and report 
data, as well as how they use data to guide their own 
decision making. 

This report is a summary of our findings and is organized 
into three sections. The first section describes what 
we learned from a series of data audits conducted in a 
sample of DC Public Schools. The purpose of the audits 
was to identify data reported by the schools, as well as 
identify information collected at the school, by whom, how 
often, where it was stored, and barriers to data collection. 
This section also provides a summary of if and how data 
are used in schools and classrooms. Key findings include: 

* Significant chaiienges to data quaiity. 

During the study period, there was a lack of universal 
practice and oversight by the district in creating data 
comparable across DCPS schools and ensuring 
accurate information within the system. For example, 
there was no central control over student ID creation 
and no validations (automatic or hand-checked) to the 
system to guard against duplication. 

* Infrequent use of data. 

During the study period, student data was used 
infrequently by school staff. In those schools that did 
use data, the data often appeared to be used partially, 
ineffectively, and/or inappropriately. For example, while 
some schools were using formative assessment data to 
track students, others used it to drill on test items and 
teach ONLY what was tested. In addition, most schools 
were using these results to identify and focus on the 
students just below the threshold of passing. These 
unanticipated and inadvertent uses of data are clearly 
not effective in guiding good instruction for each and 
every sfudent and are, in our opinion, the result of a lack 
of knowledge about how to use data in 
meaningful ways. 



5 Across the dataset 45 percent of students attending public schools had only one year of data While 54 percent had at least two years, these data may or may not be consecutive. 

6 Please note these percentages only include those students that attended public schools for more than one year and do not contain students that attended a public school one year, a charter the 
next and an alternative school the following year (or any other plausible combination). Public schools were the focus because of the completeness of their data in the dataset. 
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In the second section of this report, we make 
recommendations for laying a foundation for fufure data 
coliection and use in DC Public Schools. 

Data collection recommendations include the following:^ 

• Establish common, standardized definitions and 
coding procedures across the school district for 
collected data; 

• Establish a set of validafion rules fhat are applied 
fo submitted data prior to formally accepting the 
data; 

• Perform stafisfical checks on data submitted by 
schools; 

• Establish a system for investigating the accuracy 
of data flagged by fhe sfafistical checks; 

• Creafe sfandards for the percent of departing 
sfudents that schools must be able to locate; 

• Conduct on-site data quality checks at a number 
of schools each quarter; 

• Determine and provide supports needed for 
schools fo ensure fhe collection and submission 
of accurate and complete information (staff, 
computers, training, etc); 

• Impose consequences on schools that do a poor 
job of collecting and submitting accurate and 
complete information. 



Data use recommendations include building a data-rich 
culture, a step that requires at least five components: 

1. leadership support; 

2. a collaborative data team; 

3. a regular time to meet 

4. a central location for data, and 

5. alignment of resources. 

Einally, in the appendix we provide descriptive, cross- 
sectional analyses of one measure that has been 
associated with school failure and for which we have 
good data— students who are overage for grade. 



7 Adapted from the Data Quality Campaign’s Survey Elements: http://www.dataqualitycampaign.org/survey/elements 
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METHODOLOGY & DATA LIMITATIONS 



To learn more about data collection and use at the 
building level, we selected two types of schools for site 
visits: (1) those that we determined were relatively data 
savvy, and (2) those that struggled with data use.® Four 
schools were invited to be part of this learning process 
and all four agreed fo participate. With one exception, 
the schools were part of a new (P)K-8 organizational 
structure in DCPS, wherein elementary schools began 
adding middle school students around 2005-06. All 
schools were minority-majority schools and all but 
one school exhibited proficiency rates of less than 50 
percent. In each school, we interviewed statf about how 
data are collected, who collects the data, and how data 
are used to inform practice. During the interviews, school 
staff shared lessons learned in establishing a "culture of 
data-use.” 

During the spring of 2009, one site visitor interviewed 
all school staff for this project, ensuring continuity 
among sites. The school principals were interviewed 
and/or additional staff such as teachers, coaches, 
and attendance officers who could provide essential 
information concerning the use of data at the school. 
Interviews with principals employed protocols designed 
to capture information concerning the school’s vision of 
data practice and how the culture of data use was created, 
the specific goals of the data practices and resources 
provided to bring those goals to fruition, efforts to build 
organizational capacity as well as barriers encountered 
in their data work with staff and, most importantly, with 
the district Other staff interview protocols covered topics 
such as conception and realization of the school’s vision 
for data use, supports attributed (and needed) for the use 
of data, probing questions about the details of frequency, 
type of and accountability of staff data use, and the ability 
of their data practice to serve the needs of all students 
well. 



In addition to these schools, information gathered from 
interviews and site visits conducted at other schools 
across DC are also included in this report. These schools 
were part of a different study we conducted and these 
visits were not specifically designed fo discuss data 
collection and usage, but rather the progress of creating 
a K-8 model. Since creating a data community is just one 
component of a well-functioning school, information from 
these visits is also incorporated within this write-up for 
the purpose of providing additional insights. 

It is important to note some of the inherent data 
limitations of drawing system-wide conclusions about 
data use or interaction with the STARS system based on 
the small, somewhat homogenous nature of this sample. 
For example, high schools and middle schools may 
use STARS more frequently than elementary schools 
for schedule development and attendance tracking. In 
addition, we recognize that different school staff interacts 
with DC STARS in various ways depending on their role. 
For example, registrars interact in a different way than 
master schedulers or teachers or principals. 

Finally, the information presented and the conclusions 
drawn are a reflection of data collected at a past point 
in time, rather than present day DCPS. This study should 
not be approached as a survey of current practices in 
the district, but as an account of the data practices and 
limitations the district faced in recent years that may yield 
some useful lessons for DCPS and other urban school 
systems moving forward. 



8 Determined through our previous and ongoing work in DC middle schools. 
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SECTION I 



AT A GLANCE: 

DATA COLLECTED BY THE SCHOOLS 




DC CAS: The District of Columbia Comprehensive 
Assessment System (DC CAS) was introduced in 
2006 as a replacement for the Stanford 9 assessment. 
DC CAS was modeled after the Massachusetts 
Comprehensive Assessment System (MCAS). DC 
was hoping to replicate Massachusetts' success: 
Massachusetts' is both renowned for its assessment 
system and respected for its students’ educational 
achievement, as measured by the NAEP. Although 
MCAS is still the stated ideal, today's DC CAS looks 
more like the Stanford 1 0 assessment 

Sfudents in grades K-8 fake fhe math and reading 
portions of fhe DC CAS. A science portion is 
administered to students in grades 5 and 8 only. The 
DC CAS consists of multiple choice and constructed 
response questions (open-ended questions, to which 
students respond in writing) designed to test DC 
Learning Standards. Students are rated at one of four 
levels: below basic, basic, proficient and advanced. 
The reading portion tests three strands of sfandards: 
vocabulary, informafional fext, and liferary fext. The math 
portion tests 5 strands of standards: number sense and 
operations; patterns, relations, and algebra; geometry; 
measurement; and lastly, data analysis, statistics 
and probability. The test is given in April and is highly 
anticipated, as test results provide an indication of 
whether students are making progress toward mastery 
of state content standards. They do not, however, count 
toward grade level promotion/retention or graduation 
decisions. 

DC BAS: The DC BAS tests are the interim 
assessment/benchmark portfolio in DC and are 
adminisfered to students in grades 3-10 four times 
throughout the year. The first test of fhe year provides a 
baseline; the other three tests measure student growth 
on the same grade- level learning standards. DC BAS 
tests the same standards on each test (as opposed to 
only the standards taught between tests) allowing for 
a measure of growfh overtime. The tests mirror the DC 
CAS in format and substance. 

Reading Level Assessments: The schools used 
either DIBELS or Running Records to record student's 
reading mastery levels. The Dynamic Indicators of Basic 
Early Literacy Skills (DIBELS) and Running Records 
are vocabulary and reading tests that measure early 
literacy skills from grades K-6. They are designed to 
be a short (3-4 minutes) but thorough assessment of 
a sfudent’s vocabulary acquisition and reading fluency 
and comprehension. Students are asked to individually 
read a list of grade-appropriafe vocabulary words and a 



reading passage out loud as a teacher records students' 
errors, self-corrections, substitutions of words, and other 
information. Teachers using DIBELS use a palm pilot 
to record this information while teachers conducting 
Running Records record this information on paper. 
These assessments are used to keep track of students' 
reading progress and to monitor their reading levels. 

Accelerated Math: Accelerated Math is a computer 
program used in grades 1-8 to help students learn and 
build upon math content, knowledge, and skills. It uses 
an ongoing assessment to determine where students 
need further practice and provides them with such. 

Attendance: Attendance is recorded at each school, 
either by a teacher or office staff member. Middle 
schools are fo report attendance rates each period as a 
district mandate, but do not do so as a universal practice. 
There is no universal policy regarding the tracking of 
tardiness, excused absences, and early dismissals. The 
default designation in the program is “present” and 
schools are incentivized to have high attendance rates - 
from funding to district accolades; attendance is almost 
always part of the School Improvement Plan by which 
the principals are held accountable. 

Classroom Grades: PK-6th grade teachers keep a 
paper grade book and record report cards on carbon 
papered report cards. These hard copy-only reports 
are kept in students' cumulative files. In one school, 
teachers had electronic grade books. In all other schools 
an electronic grade book was not something the staff 
desired or felf fhey needed. In these schools, middle 
grade teachers keep grades in a paper grade book and 
enter mid- and end-of-quarter grades in STARS, where 
the progress reports and report cards are generated. 

Behavior data: All schools visited keep track of 
behavior by filling ouf paper referral forms that were 
submitted to the principal. The forms vary but usually 
ask for name of student, grade, date, teacher, who 
the referral was inifiated by, the discipline infraction 
and the resolution (detention, parent phone call, etc.). 
Completed forms are kepf in the students' record, in 
a centrally located binder or a file cabinef or by fhe 
teachers, as set by the school policy. This information is 
not recorded electronically and when asked if recording 
it electronically would be helpful, some indicafed fhey 
would (but had no plans of using the information); still 
other schools indicated a lack of resources to input the 
information electronically into STARS. 
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SECTION I 



DATA BEING COLLECTED 
AND UTILIZED BY THE SCHOOLS 



Most data collected within the schools appeared to 
be formative assessments, although school personnel 
collected attendance and behavior information as 
well. Schools received demographic information from 
the district as well as the students’ previous year's 
assessment data (DC CAS scores). 

Data are supplied to the district and received from the 
district through a Student Information System (SIS)/ 
data warehouse software called Student Tracking and 
Reporting System (STARS).® DCPS purchased STARS 
in the spring of 2004 and began a three-phased 
implementation for rolling out the software. Starting first 
with a pilot school, the district then provided the software 
to elementary and finally middle and high schools. With 
the new system, DCPS hoped “to change the paradigm 
for school accountability through the implementation 
of district-wide standards for curriculum and courses, 
enrollment practices, attendance, scheduling practices, 
graduation requirements, report cards, and transcripts.”'® 
DCPS is still working to achieve this goal. 

The only data that schools discussed viewing or 
downloading via STARS were student demographics. 
Schools uploaded attendance and middle school grades 
to the system, as well as recorded student withdrawal 
information. All other data collected at the schools were 
housed within other programs or hard copy only. We are 
uncertain if these other programs are compatible with 
STARS. 

ASSESSMENTS 

DC CAS: DC CAS results from the previous April test 
administration are delivered to schools in mid-fall of the 
following year. Several types of hard-copy reports are sent 
to the schools: individual student reports (Figure 1 ), class 
reports of proficiency, a report showing a breakdown 
of all students’ performance on each test item, and 
proficiency rates disaggregated by demographics. The 
only information provided for an individual student is 
shown in Figure 1 . 



According to our study participants, staff members rarely 
engage with DC CAS data and for several reasons. First 
and foremost, these kinds of tests are not designed 
to be useful for schools in making adjustments to 
classroom instruction. For example the deepest level of 
disaggregation of the DC CAS test information is the 
strand (3 strands for reading and 5 for math; see Figure 
1). Thus, teachers are unable to determine with which 
specific skills students are struggling, making it difficult 
to manipulate the data to answer specific questions on 
student performance or plan specific directions in which 
to teach because the data do not allow that level of detail. 
Charts indicating the average percentage of students 
proficient in math and reading over time are often hung 
on the wall, but largely forgotten during discussions or 
decision-making. 

DCBAS: Schools depend on DC BAS scores as their 
main data source. The DCPS data are disaggregated by 
strand and learning standards and stored in a program 
called Discovery ED, which allows schools to compare 
themselves against the district average and other 
schools. Using Discovery ED, staff can also measure 
student growth in several ways: against average student 
growth in the district; against a set trajectory of “normal 
or expected growth;” within a class; and/or with a 
particular student. However, such growth is only in the 
aggregate; measuring growth or set-back on particular 
learning standards is difficult and requires information to 
be pieced together from one test to the next. Because 
interim assessments are only given every six weeks, mid- 
course corrections to topics of further teaching tend to 
only be done on this timeline. 

Reading Level Assessments: If using DIBELS, 
student’s errors, self corrections and other stumbles while 
reading a passage or list of vocabulary words are fed into 
a computer program, which issues a “grade:” red (“alarm;” 
in need of reading intervention); yellow (“basic;” in need of 
reading intervention); or green (“on grade level;” no need 
for intervention). The program recommends intervention 



9 Information from this audit was further verified by District of Columbia Public schools and public charter schools enrollment census, October 5, 2007 by Thompson, Cobb, Bazilio & Associates, 
PC http7'/www.seo.dc.gov/seo/frames.asp?doc=/seo/lib/seo/information/school_enrollment/2007- 2008_Final_Audit_Report.pdf 

10 Pg 2; District of Columbia Public Schools: Student Information System Implementation to Support Transformation in Accountability: 

http://www 03.ibm.com/industries/global/files/dcps_services_case_study_f inal.pdf ?re=education&sa_message=title=district_of_ columbia_public_schools 
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FIGURE 1 : DC CAS STUDENT REPORT 




Source: http://www.k1 2. dc.us/offices/oda/doc/DCCAS_Overview_Brochure_FI NAL_March6.pdf 



The Senior Urban Education Research Fellowship Series, Voiume I - Fall 2010 



SECTION I 






SECTION I 



DATA BEING COLLECTED 
AND UTILIZED BY THE SCHOOLS 



strategies for teachers to use with each student, based 
on his or her strengths and weaknesses, along with 
worksheets and lesson plans to target areas in need 
of improvement. Running Records requires teachers to 
analyze the data and based on the types of mistakes the 
student is making, set up intervention strategies. These 
assessments are time consuming because they are done 
one-on-one with the teacher and tend to only be done a 
few times a year, if not only at the beginning of the year. 

Accelerated Math : At the beg i nn ing of the year, students 
take a diagnostic test called STAR (not to be confused 
with the SIS used at the district), which the teacher then 
reviews. Based on the students' performance, fhe teacher 
assigns the appropriate Accelerated Math lessons. The 
computer tracks students' progress in each lesson and 
responds to students' struggles. 

OTHER DATA 

Attendance data: According to the staff we interviewed, 
attendance data are rarely part of conversations about 
students or their achievement Only one of fhe schools 
we visited posted attendance data and it was used as 
an outcome rather than an additional piece of input 
information. Reports are run from STARS weekly so 
schools can ensure attendance was taken every day. 
Schools with a staff member designated to work on 
attendance appear to have a well-run system of using 
the data effectively to identify students that have been 
absent for more fhan 3 to 5 days, notifying parents, 
having a conference and attempting to figure out how to 
get the child to school more regularly. 

Classroom Grades: We did not find evidence that 
grades are used as a measure of progress or success in 
data analysis by the schools. 

Behavior Data: As mentioned earlier, behavior data 
consists of paper referral records. These records are 
mainly used as backup documentation when meeting 
with a parent about unsatisfactory behavior or when 
justifying a suspension. These data are not used as a 
measure of progress or success in dafa analysis by the 
schools. 



Demographic data: We also found no evidence thaf 
demographic data are used at the schools to examine 
patterns in the other data collected (e.g. test scores by 
race/ethnicity gender, etc.). 

Student withdrawals: As mentioned earlier, there are 
no guidelines for systematic collection of this information 
and we did not find that they are used as a measure of 
progress or success in data analysis by the schools. 

DISCUSSION 

As we have learned, having access to data is certainly not 
the same as using data, and using data is not synonymous 
with using it effectively to guide and refine instrucfion. 
Esfablishing the foundation to facilitate and support the 
use of data was needed, at least to some degree, in all of 
the schools that we visited. In cases where schools used 
data, such as the DC BAS, the data appeared to be used 
partially, ineffectively, and/or inappropriately. For example, 
some schools were using the DC BAS to track students, 
others to drill on test items and teach ONLY what was 
tested but most were using these results to identify and 
focus on fhe students Just below the threshold of passing. 

These unanticipated and inadvertent uses of these data 
are clearly not effective in guiding good instruction for 
each and every student and are, incur opinion, the result of 
a lack of knowledge about how to use data in meaningful 
ways. Based on our observafions and understanding of 
the extant literature and data tools, we recommend that 
DCPS take a number of steps to build a data community 
to collect accurate data and use these data to further 
their goals of creating educational experiences that meet 
the needs and challenge the strengths of all students. We 
discuss these recommendations in further detail in the 
section that follows. 
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A tool such as identifying students that are at risk 
of dropping out is only as effective as the data are 
valid, reliable, and used to inform policy and practice. 
The schools we visited were at varying stages of 
incorporating a strategy for data use, such as an early 
warning indicator. While approaches to data use differed 
across schools, the necessary precursors for successful 
use are fundamentally the same. As illustrated in Figure 
2, this next section discusses the key precursors to 
developing an effective, data-driven decision making 
process and data-rich culture and how the intersection 
of federal, state, and district policy can and do effect the 
data collection and use in a school and classroom. We 
touch briefly on the important role of these entities in 
ensuring that schools have the opportunity to create the 
essential conditions to use data effectively. Drawing from 
the literature on data use, we also highlight examples 
from our site visits in this section. Specifically, we discuss 
the type of foundation a school needs to collect, analyze 
and use data to guide instruction, make decisions, and 
plan for and implement a comprehensive improvement 
process using data. Lastly, we draw attention to what the 
research says about the process of how schools should 
go about using data effectively, providing detail regarding 
where the schools we visited were in this process. 

ENSURING DATA QUALITY 

The first step in building an effective data community is to 
establish systematic procedures to collect reliable, valid 
information that informs practices. The district and state 
are critical to this step, often providing resources and 
developing quality assurance policies. While this report 
does not go into great detail about the roles of district, 
state and federal stakeholders, as outlined in Figure 2, 
we will briefly discuss the importance of their presence in 
establishing high quality data collection standards. 



The Role of Federal, State, District Policy 

There has been much focus at the state level around 
building state longitudinal databases, but little to no 
mention of how schools will use the information collected 
or how the information should be collected. Since most of 
the data that populate state longitudinal databases come 
from schools, establishing and implementing systematic 
collection procedures is key to having accurate and 
usable data. Thus, initiatives and recommendations at 
the state and national level must expand their designs to 
involve not just districts, but schools and most importantly, 
teachers (see Figure 2). 

Using data to improve student outcomes requires 
accurate data collection procedures throughout the 
education pipeline, but equally important, it requires the 
collection and applications that reflect what principals 
and teachers need to help their students succeed. 

Invalid or unreliable data provided by some schools and 
districts is an issue in some states and these inaccuracies 
will continue if quality assurance procedures are not 
developed, implemented, and tested.” A well-designed 
and well-implemented state data audit system will help 
identify such areas of inaccuracy.'^ According to self- 
reported data (to the Data Quality Campaign), DCPS 
and OSSE do not yet have a process for conducting data 
quality audits but there are plans to develop one.'^ This is 
not surprising given the issues we highlight below. 

Our interviews provided information that outlines how 
district and/or state policy (or lack thereof) in DCPS 
has sometimes hindered the use of data effectively in 
the district schools. For instance, there have been several 
critiques of the STARS program and DCPS’ use of It. 
STARS is the SIS for the DC Public schools only. Charter 
schools in the district do not universally use or have 
access to the program. As such, duplication of students 
across DCPS and charters has been problematic 



1 1 Pinkus, L (2006). Who's Counted? Who's Counting? Understanding High School Graduation Rates. Alliance for Excellent Education. Washington, DC. 

12 Pinkus, L (2006). 

1 3 Data Quality Campaign Survey Results, www.dataqualitycampaign.org/survey 
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FIGURE 2: INTERSECTION OF NATIONAL, STATE, DISTRICT, AND SCHOOL 
IN CREATING A SYSTEM OF EFFECTIVE DATA USE 
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because there is no way to verify if a student is enrolled 
in DCPS as well as a charter school. As students move 
from charter to DCPS, or vice versa, student Identification 
numbers may also duplicate because there is a lack of 
central control of the assignment of student identification 
numbers— school personnel are using and are authorized 
to assign different ID numbers. 

According to the administrators and teachers we 
interviewed, the system does not require anything 
more than a last name to match a student to an already 
assigned student ID number. This is problematic because 
if there is a slight change in the spelling of the name 
(either via human error or just a previous or current 
incorrect record) or a name change, the student will be 
assigned a new ID number. A school can also assign an 
identification number to the student that was provided by 
the parent (and could be incorrect). There are no current 
validations (automated or hand-checked) to the system 
to ensure duplication of IDs does not occur.''' 

This need of central control over student identification 
numbers may have greatly contributed to our lack of 
longitudinal data. We may have had a much higher 
percentage of students in the dataset over time, but 
students may have had different identification numbers 
over time. 

One common approach to addressing such issues is using 
birth date to identify duplications. This is not effective 
with the DCPS data, which lacks validation for dates of 
birth. The administrative records received from DCPS 
(via Harvard University) indicated several erroneous and 
obviously incorrect dates of birth for the grade level- 
students much too young or old for the grade level or the 
same student with multiple dates of birth. There are no 
current acceptable ranges for dates of birth in the DCPS 



data system and thus no flags are raised when obviously 
erroneous birth dates are entered. This is particularly 
problematic for developing an early warning indicator, 
because overage status is an important component of 
academic risk.'^ 

A lack of universal practices and processes laid out by 
the district and/or state also pose a challenge to creating 
data comparable across DCPS schools and ensuring 
accurate information within the system. For example, 
there is no standard way that schools record attendance. 
Some collect attendance records from teachers every 
day, some every week. Some middle grade teachers take 
attendance every period, others do not. Some teachers 
use A for absent and T for tardy, EA for excused absence, 
and UA for unexcused, while other teachers use check 
marks, hashes, slashes, and so forth. 

There is also a lack of understanding over what is the 
"official record”; the STARS report, the manually kept 
record by teachers or the attendance card that the court 
system recognizes. Because students are defined, by 
default, as “present,” unless marked absent, failure to 
input absences results in misleading information and 
favorable average daily attendance rates that the schools 
are not likely to correct. 

Another example of the need for universal practice is 
recording withdrawal rates. Withdrawal dates are not 
captured as the students' last day at the school (but 
rather as the day the withdrawal is actually entered, the 
day the parent tells the school the student has withdrawn, 
the day another school lets them know that they have 
enrolled in their school, or any other date in between) and 
as such, a student will appear to be present after having 
withdrawn. 



1 4 According to DCPS, DC STARS checks the following fields for potential existing matches before a new student ID Is generated: Last Name, First three characters of first name, date of birth, 
and gender. A listing of existing potential student matches is generated in the system based on these criteria and users receive a warning message to review the list to avoid multiple IDs being 
created. Since 2007, principals have also received weekly data corrections reports that outline students that could potentially have multiple IDs. The administrators, teachers, and counselors 
interviewed, however, did not seem to have complete knowledge of this process at that time. 

1 5 With little missing data on birth date and additional data points, we were able to construct an overage measure with confidence and examine it as part of this project. 
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Recommendations to ensure that 
collected include the following:^® 

• Establish common, standardized 
definitions and coding procedures 
across the school district for collected 
data; 

• Establish a set of validation rules that 
are applied to submitted data prior to 
formally accepting the data; 

• Perform sfafistical checks on data 
submitted by schools; 

• Establish a system for investigating the 
accuracy of dafa flagged by the 
statistical checks; 



valid, reliable information is being 

• Create standards for the percent of 
departing students that schools must 
be able to locate; 

• Conduct on-site data quality checks at 
a number of schools each quarter; 

• Determine and provide supports 
needed for schools to ensure the 
collection and submission of accurate 
and complete information (staff, 
computers, training, etc); 

• Impose consequences on schools 
that do a poor job of collecfing and 
submitfing accurate and complete 
information. 



1 6 Adapted from the Data Quality Campaign’s Survey Elements: httpL//www.dataqualltycampaign.org/survey/elements 
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ESTABLISHING THE FOUNDATION 
OF A DATA COMMUNITY 

Based on our understanding of the literature, we posit 
that establishing a data-rich culture requires at least five 
components: (1) leadership support, (2) a collaborative 
data team, (3) a regular time to meet, (4) a central 
location for data, and (5) alignment of resources. Each 
component is described below, along with pertinent 
information from our site visits. 

Leadership Support 

The leadership of the school sets the tone, priority, 
and mission for school and staff. A leader well trained 
in using data— or just one supportive ot its use— can 
create a ripple ettect within the school. The leadership 
also sets the priority ot collective responsibility for using 
data by ensuring that the other four components (e.g., 
collaborative data team, time to meet) are in place.'^ 
State and district policy are important in ensuring that 
leaders at the school level are supported in implementing 
a data community. Nationally, advocating and expecting 
leaders to use data in planning, running, and carrying out 
the business of education is also equally important. 

It is no surprise that the schools we visited with stable 
leadership tended to have more established processes 
in place and were positioned to begin addressing critical 
questions using data. The schools in which the principals 
had been in place for several years were able to establish 
a data use program and build upon and refine procedures 
and processes for not only collecting data but tor using 
the information as well. These principals were more 
successful creating teams of staff with a variety of 
skills, providing the means necessary for professional 
development when needed, allocating resources such 
as time and equipment to data use, and establishing a 
schedule that allowed staff to meet collaboratively on a 
regular basis to discuss data. 



Schools with newer principals were still Just gaining 
their “sea legs” and understanding their new roles and 
responsibilities-even if they were veterans to the field but 
new to the school. Many times, these new principals were 
working with a schedule and budget that they had not 
created, were unaware of the many different software, 
hardware and/or other resources available, and were still 
in the process of assessing the skills of their staft and the 
necessary resources. 

Creating a Collaborative Data Team 

Creating a team is probably one ot the most important 
foundational components to creating a data-rich 
culture.'®’'® Through a collaborative team process, data 
should be used frequently and regularly to discuss not 
only student outcomes, behaviors, attendance patterns, 
and results (formative and summative), but also teacher 
inputs. Instructional reflection, such as the way in which 
the material was presented, the order, the amount ot time 
given, the questions asked, and what was asked of the 
students are all also critical “data” to consider. One way to 
deconstruct teaching practice is to observe one another 
or video tape classes and get together to discuss, thus 
adding data points by which to better understand the 
teaching and learning process. To successtully analyze 
and collect data and ask and answer difficult questions, 
staff need a collaborative, safe place to discuss one’s own 
successes and— equally as important— shorttalls; without 
trust and collaboration, moving the conversation to this 
latter list will not be constructive. Creating such a place 
requires more than inviting people to a meeting. Creating 
a collaborative team requires a strategy and a great deal 
of effort. Figure 3 illustrates critical components of a 
collaborative team 



1 7 Parker Boudett, K., City, EA, Murnane, R.J. (2007). Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning. Harvard Education Press: Cambridge, MA. 

1 8 Love, N (2002). Using Data/Getting Results: A Practical Guide for School Improvement in Mathematics and Science Education. TERC: Cambridge, MA. 

1 9 Parker Boudett, K. et al. (2007). 
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FIGURE 3: CREATING A COLLABORATIVE DATA TEAM 




Source: adapted from: Five Dysfunctions of a Team by Patrick M. Lencioni 



Establishing trust: Trust is the foundation on which a 
great team is built. It encompasses trust in each other and 
trust in the data. Without trust, both from the leadership 
and each other, team members will not be encouraged 
to grow. 

Acceptance of conflict: Conflict can be a negative and 
a positive within a team. If members do not feel open 
enough to speak their mind, no real progress will be made. 
But if there's a lack of trust and members speak their 
mind in a disrespectful way, progress will be hindered as 
well. Conflict allows the team to mature, but only if the 
other pieces of team are established along the way. 

Foster Commitment: Commitment needs to be 
established within the team members and/or nurtured 
throughout the process to avoid burnout or bored-out. 



Embrace Accountability: Holding each other 

accountable to the process and to the students creates a 
collective responsibility wherein all members are equally 
answerable. 

Driven by Results: the Cheshire cat in Alice in 
Wonderland said it best: “if you don't know where you're 
going, any road will do.” Unless goals are established by 
the team around what they want to accomplish with using 
data and the team strive for those results, the time will 
be wasted. 

In the schools we visited, we found fhat staff used 
the scoring of constructed response items on the DC 
BAS as a way to establish a team around data. Every 
six weeks, all teachers (or in some schools, all staff), 
including elective teachers would get together and score 
the constructed response questions. Scoring in a group 
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ensured collective understanding and responsibility in 
that all staff knew where sfudents were doing well and 
struggling. The staff was fasked fo build on any identified 
weaknesses in students' skill sets and incorporate the 
areas of strength into their class regardless of subject 
area. Other schools mixed teachers and staff of differing 
grade levels and subject areas by design to begin fo 
build vertical alignment among the grades and share 
information across staff. For example, by grading 4fh 
grade reading questions, the PE teacher was aware of 
what was expected of a 4th grader in reading, and a 
2nd grade teacher would become knowledgeable of 6th 
grade math expectations by grading those responses. 

One school created a professional learning data team 
consisting of the counselor, several teachers and the 
Assistant Principal. The team met to analyze, reflect and 
plan using data with all staff, but the team also hosted 
workshops for the staff to build skills where needed 
and develop data analysis capacity among the staff. 
The team met over the summer and created a calendar 
of professional development offerings and scheduled 
meeting times and dates. These offerings were based 
on the skills of the teachers; what the team could teach 
others as well as what the teachers needed. Establishing 
such professional learning communities could be a highly 
effective collaborative data team arrangement. 

Establishing a Regular Time to Meet 

Collecting and analyzing data takes time and focus. 
Without time set aside for this work to take place, 
most schools will not get farther than collecting the 
information, if fhat far at all. Instituting a regularly set time 
for fhe team, and/or sub-team to meet to collectively 
review, reflect, question and plan using fhe dafa most 
effectively ensures data are used strategically to make 
decisions.^° If the data team is a subset of staff, regularly 
scheduling time for the data team to meet with the full 
staff is also needed. Protocols for asking and answering 
questions should be established and goals should be set 
and reviewed each month. 



As a district mandate, all schools had time from 8:00-8:40 
a.m. to use for staff meetings, collaborative work, and data 
discussions. Pew schools built in additional planning time 
to be used as collaborative time when teachers could get 
together to discuss data, share lesson plans, plan units, 
talk about students, and discuss curriculum. There were 
several reasons for this. Some schools were unclear 
about how fo use this collaborative planning time and it 
often ended up being wasted time from the leadership's 
perspective or not worth the effort of incorporating it 
into the schedule. However, some schools had such a 
limited number of staff— without corresponding reduction 
in the number of classes— that squeezing out any more 
time in the schedule for teachers to get together was 
impossible. The district mandates that teachers are 
assigned to schools based on enrollment, but the district 
does not take into account the need at the middle school 
level for additional classes. Schools with particularly low 
enrollments were most affected by this policy. Therefore, 
because of such fypical fime limitations of 30-40 
minutes, data discussions were quick, infrequent, and not 
very substantive. Discussions about data coincided with 
the DC BAS, taken by students every 6 weeks. These 
discussions focused on how many students scored 
proficient, how the school compared to other schools in 
the district, and the areas teachers needed to re-teach. 
Decisions using data tended to be made by one or two 
staff members and relayed to the team during this time 
instead of the team responding to the information and 
making informed decisions as a group. 

Policies at the district, state and national levels can be 
altered to support the use of data in schools. Creating 
aligned systems for education within and across levels 
of education and providing resources and materials 
(including relevant data tools and applications) aligned 
with common standards would provide a beacon toward 
which teachers can steer their instruction, assessments, 
and students' performance. 



20 Parker Boudett, K. et al. (2007). 
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Securing a Central Location for Data 

Data that are not linked and/or integrated across 
programs provide an incomplete picture of student 
progress. When information must be pieced together, it is 
rarely done. Warehousing demographic information with 
attendance, behavior, grades, and achievement measures 
will allow staff to construct more meaningful infervenfions, 
assess their own success in working with different 
learners, and identify areas in need of improvemenf from 
a students’, teachers' and schools’ perspective.^' Ideally, 
this central location would be electronic and web-based, 
accessible 24-7 by all from any location with Internet 
access. Software should also be easy to use and staff 
should be fully trained to use the software efficiently. 

However, the schools we visited did not always have the 
resources to create an electronic “data room,” so they 
created a physical one. For example, a school created 
a data room that they use to showcase and display test 
data, attendance, reading levels, and similar data. In 
addition to teachers using the room, students and parents 
were also given access so they were knowledgeable 
about their own or their child’s data. Students especially 
enjoyed bringing their parents to the room when they had 
made growth over time. This allowed data to be used, 
and conversations to be had even when the team was 
not meeting. In another school, all teachers had a data 
notebook to centrally organize the data they received 
about their students at the beginning of the year, the data 
they collected over the year, as well as any data-related 
materials they received throughout the year (e.g., tools, 
guides, workshop materials). 

Aligning Resources 

Resources may be defined broadly, but are absolutely 
necessary. This includes human capacity (such as well 
trained staff), training opportunities (such as workshops 
and courses) and materials (such as computers and 
software). Often, schools, including those in DCPS, 
tend to have many resources but they may or may not 
be aligned to the priorities of the school, district, and/ 



or state. Therefore, resources are often used piecemeal, 
rather than strategically toward a common goal. 
Leadership and/or Data Teams that establish using data 
as a priority need to outline the work ahead and make 
certain that staft have the resources they require (e.g., 
the skills necessary to collect, analyze, and plan using 
data, software, hardware, and other supplies to complete 
the work). Districts and states that align their priorities 
tend to have an easier time aligning resources and 
therefore provide what is needed to the schools to meet 
such goals. 

Staff 

During our site visits, we found that the schools that 
struggled most were those with the leanest staff. They 
were also the schools that had data quality concerns 
because there was no one person designated to 
regularly collect the information needed. Alternatively, 
a process was needed to ensure that multiple staff 
collecting information did not translate into multiple ways 
of collecting the information. We identified two critical 
positions that facilitated data collection and use: 

• Attendance Officer: The attendance officer enters 
attendance every day in STARS and then informs 
teachers which students are absent, tardy, and so 
forth. 

• Coaches/Data Team Leaders: These staff usually 
held professional development sessions with 
teachers, topics consisting of system require- 
ments on the computer, how to use the poster 
maker as well as how to break down the results 
of the students DCBAS, Accelerated Math, or 
other data elements. They work with teachers to 
create long-term plans and goals and then back 
map what students need to learn along the way 
in order for them to reach those goals. 



2 1 Love, N., Stiles, K., Mundry, S., DiRanna, K. (2008). Data Coaches Guide to Improving Learning for All Students: Unleashing the power of collaborative inquiry. TERC: Cambridge, MA. 
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Training 

Several school leaders received DataWise training. This 
training helped to standardize language used at the 
school around data, as well as provide a process for 
collecting and analyzing test score data. Schools with 
time and staff provided in-house training on the process 
and began to build staff capacify to work with data. 
As a district mandate, staff need to be trained in the 
procedures and processes used around data use in order 
to be truly effective. 

Materials 

Those we interviewed expressed a lack of confidence 
in the current district software and hardware. There is 
a need for training to utilize the full capabilities of the 
resources the schools have been offered. 

One resource making data use easier is a student 
information system by which to store and organize student 
information. The schools were confident there were other 
capabilities within STARS that were not being used that 
might be helpful, but they had not been trained on any 
part of the system other than to record attendance or 
look up student demographic information. For example, 
school coaches sat with principals for extended periods 
of time attempting to schedule classes in a building that 
holds elementary and middle school students; STARS 
has a scheduling function but no one knows how to use 
it Currently, STARS is not seen as a school tool, but as 
a district mandate where attendance data and middle 
school grades must be uploaded. 

The schools also showed wide variability in their levels of 
comfort and knowledge in using many of the products 
purchased by the district or individual schools, such 
as Discovery Ed or Accelerated Math. As such, many 
programs are not being used as they were designed. 

The case is similar with hardware. Because all buildings 
are not networked, much of the data kept by teachers is 
on their own computer. 



As indicated, there were five major components needed 
to take the schools’ use of data to further or full 
implementation: (1 ) leadership support, (2) a collaborative 
data team, (3) a regular time to meet, (4) a central location 
for data, and (5) alignment of resources. The schools we 
visited exhibited varying levels of establishing each but 
would greatly benefit from focusing on these precursors 
in attempting to better solidify their practice around the 
use of data. 

USING DATA EFFECTIVELY 

After creating a space and culture by which to use data 
and ensuring that the data are accurate, reliable and 
valid, what should a school do with the information? 
What information should be collected? What information 
would be helpful to analyze? How should information 
be analyzed? How often? Once analyzed, what should 
staff do with the results? These, and many others, are 
questions the staff will begin to ask themselves as they 
forge ahead in this process. Below, we describe how 
schools we visited fared in terms of the data-use process, 
as outlined by the “model” they have been trained to use: 
Data Wise. 

Prepare 

Preparing to use data requires an inventory of what 
data are currently being collected as well what data 
are needed.’^^ At the schools we visited, the amount of 
data collected at the middle school level is very slim in 
comparison to that collected at the elementary grades. 
For example, although there are middle school equivalents 
to DIBELS and Running Records to assess reading 
levels such as Scholastic Reading Inventory, Fountas and 
Pinnell, none of the staff we interviewed reported using 
these to assess older students. If the district has not yet 
vetted or determined resources to be used at the middle 
school level, it is not surprising that schools have not 
purchased any software/tools/resources due to budget 
constraints. The district and schools will need to focus on 
this, along with the PD to use these resources. 



22 Parker Boudett, K. et al. (2007). 
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SECTION 2 



BUILDING A DATA COMMUNITY 



Inquire 

The Inquiry Phase of using data effectively entails creating 
an overview analysis of the data as well as “digging” into 
the analysis to the root of the problem in questionT^ Staff 
that we interviewed discussed using DC BAS data as 
their sole source of inquiry. However, other sources of 
information are also critical for examining student and 
school performance and well being, as indicated in Figure 
4. Is attendance a problem? What about staff retention? 
Test data are just one piece of a larger puzzle. It is like 
a check engine light being on or off— when the light is 
on you know there's a problem, but you're not sure what 
it is. When the light is oft, you figure things are moving 
along smoothly even though your brakes may be failing 
or your battery running low. Focusing only on test scores 
without also analyzing additional information will result in 
an incomplete understanding of the situation and even 
less understanding of how to resolve it.^“^ 



For example, what if the student was absent repeatedly 
prior to a test and they missed valuable information? 
What if they have been getting into trouble lately and 
spending time in detention or suspended instead of in 
the classroom? Should you drill them in finding main 
ideas if they got these questions wrong or try to build up 
their reading level? Or both? 

These questions and many others cannot be explored in 
isolation - or with isolated data. One suggestion tor the 
types of data to analyze and how often can be found in 
Figure 4. The best use of data comes not necessarily 
from assessment results of formal tests but from the 
strategic reflection of daily interactions with students. 



FIGURE 4: USING COMMON ASSESSMENTS AND OTHER STUDENT LEARNING DATA SOURCES 




Source; The Data Coach's Guide to Improving Learning for All Students by: Nancy Love, 
Katherine Stiles, Susan Mundry, Kathryn DiRanna 



23 Parker Boudett, K. et al. (2007). 

24 Koretz, D. (2008) Measuring Up: What Educational Testing Really Tells Us. Harvard University Press; Cambridge, MA. 
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Act 

As stated earlier, the purpose of collecting data is to use 
it as a tool to provide feedback on students' abilities to 
guide instruction. The staff we interviewed discussed how 
information was collected in the schools, but provided 
little information about how the data were actually being 
used to improve staff practice and increase student 
outcomes. 

The leadership staff across DC had received training 
from Harvard’s data use strategy, DataWise. However, 
since staff responsible for the work had not received 
the DataWise training directly, their knowledge of the 
full process was incomplete. The assessment data were 
mostly used to group students or dictate which topics 
teachers had to focus on. Using the data to engage in 
dialogue about their own teaching, for example, was 
missing. Utilizing data to build content mastery among 
students and teachers was also missing. If staff is going 
to continue to use assessments as the only data to 
examine student progress, it is important to reflect on 
student-level performance in addition to school-wide 
trends. 

The kind of action taking place in the schools we visited 
is not necessarily what one would hope for. Such a strong 
focus on test scores and increasing such scores may 
push schools to make decisions with the right intention 
but a lack of foresight to the consequences of students' 
overall educational well being. 



For example, one school used data to track students' 
academic abilities in their elementary literacy blocks 
by having the trained reading specialist work with the 
students that were close to proficiency and above, while 
the PE teacher taught the very low- level students. 
These two teachers worked closely together and the 
reading teacher supplied the PE teacher with resources 
and lesson plans, but one could argue that the low- 
level reading students needed the experienced reading 
teacher the most. From the school's perspective, there 
was more value in bumping the students just below 
the passing threshold to proficiency and increasing 
the percent proficient in the grade than investing their 
strongest teacher with the students that had the farthest 
to go to reach proficiency. In another school, a content 
specialist resource math teacher with many years of 
experience co-taught with a 3rd grade teacher because 
this was the first year that she taught a “tested grade” and 
needed help in teaching the students testing techniques. 
One could argue that other teachers in the school could 
have used her support as well. 

The schools we visited would benefit greatly from a 
district and/or state strategy outlining the purpose of 
using data, providing the leadership support needed, and 
assisting the schools in creating a team by providing the 
time and resources and training to do this work, as well 
as a central location to collect, analyze, and house the 
valuable data being generated and utilized. 
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CONCLUSION 



When asked about a long-term vision for future data use, 
the schools we visited did not express an ideal much 
different from where they currently are. The schools had 
certainly accomplished much. In a district where most 
everything is still kept in hard copy only— grade books, 
report cards, even attendance records as required by the 
court— establishing procedures to collect and analyze 
data electronically is a mountain to move. 

However, schools still have the rest of the mountain 
range to go in terms of DCPS’ likelihood and readiness 
to create and use an Early Warning Indicator. There are 
several concerns that must be addressed before such 
a tool can be effective in identifying and supporting 
students at risk of dropping out of school. 

The first issue is the quality of the data. Several critical 
measures to an early warning indicator-date of birfh, 
attendance, grades, withdrawals- are being collected in 
different ways at different schools, not allowing for cross- 
school comparisons or data to be rolled up. Measures 
such as date of birth require internal checks and 
parameters instituted to ensure erroneous information is 
not included in the dataset Some of these data, such as 
grades, are not required to be entered electronically and 
are therefore missing from the database. 

The second issue is the lack of a sufficient longitudinal 
dataset due to practices around issuing student 
identification numbers and the inability to follow students 
from public to charter schools. Because students may be 
assigned more than one identification number, creating 
a database that follows studenfs over time is difficult. 
With the large number of charter schools in the District, 
and more being added every year, the district will need 
to facilitate a way to identify the schools students are 
attending regardless of whether it is public or charter. 
Otherwise students get “lost” in the system and are not 
counted as part of the longitudinal dataset when they 
attend a charter school. 



The third issue is the foundation of a data community within 
the schools. Schools and staff have more data to work 
with than ever before. However, they are still struggling 
to obtain the foundational pieces needed to really be 
able to utilize these data. One of the key components 
missing was the time to collaborate. Teachers and staff 
had liftle time - or had no real impetus to make the time 
- to review, reflect, or plan using the data. Teachers not 
having much time to get together to discuss their practice 
meant that while they were a “team,” they were not a 
collaborative team; they were not learning, growing, or 
improving by reflecting on the data as a group. The other 
key element missing were resources aligned to such 
data work: training and assignment of staff dedicated to 
collecting and analyzing data. 

Lastly, schools will need to push forward in collecting a 
comprehensive set of data that best helps them support 
their students. Observing assessment data every six 
weeks in isolation means that mid-course corrections 
of subject matter, changes in teaching strategies, and 
refinement in resource allocation will only occur six times 
a school year, at most. Struggling students need facile 
teachers with the ability to assess their teaching and 
students’ learning daily in order to accelerate learning. 
Until then, “using data to drive instruction” will continue 
to be a misnomer, one that is spoken but not really 
accomplished. 
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TOPIC IN FOCUS: AN EXAMINATION 
OF STUDENTS OVERAGE FOR GRADE 



INTRODUCTION 

The purpose of an early warning indicator is to identify 
students at risk of tailing school in time to effectively 
intervene. One of the strongest predictors of high school 
dropout is being overage for grade, defined as a year (or 
more) older than grade-cohort peers. Overage for grade 
is often a proxy for grade retention, whereby students are 
held back a grade due to poor academic performance. 
The assumption is that another year of development and 
another year learning the same material, possibly with 
additional supports, will prepare struggling students to 
successfully transition to the next year. While research 
does not bear out this assumption, being overage for 
grade in middle and/or high school is a strong predictor 
of disengagement and dropping out.' In fact, dropouts 
are five times more likely to have repeated a grade than 
their peers, and students who repeat two grades have 
nearly a 1 00 percent probability of dropping out.^ 

In this brief, we examine overage students in the District of 
Columbia as a warning signal for significant performance 
issues and dropping out of school. Specifically, we use 
DC Public Schools (DCPS) administrative data for 6-9th 
graders in 2007-08 to identify students overage for 
grade, and then examine their academic and demographic 
characteristics, as well as the characteristics of their 
schools and communities. 

MEASURING OVERAGE 

Because DC does not have a birthday cutoff for 
enrollment,'^ we calculated OVERAGE'' using non- 
rounded numbers and creating dichotomous variables 
(overage— Y/N) for each grade level: 

1) A sixth grader is overage if s/he is 1 2 years or 
older during the first half of the school year. 

2) A seventh grader is overage if s/he is 1 3 years or 
older during the first half of the school year. 

3) A eighth grader is overage if s/he is 1 4 years or 
older during the first half of the school year. 

4) A ninth grader is overage if s/he is 1 5 years or 
older during the first half of the school year. 



In the DCPS administrative data, there was minimal 
missing data for date of birth. However, there were 
inconsistencies in values, including years that were 
obviously much too early or much too late for middle 
grade students. We were able to clean up these data 
errors to appropriately calculate age using the following 
steps: 

Where the student had at least two DOBs 
(e.g. one DOB for 2006 and another for 2008): 

1) If the year and month matched, but the day 
differed, the student was assigned the DOB 
with the earlier date (e.g. 09/06/1981 and 
09/05/1981 , 1 changed DOB to 09/05/1981) 

2) If month and day were the same, but the year was 
different, the variable was not changed. 

3) If date and year were the same, but the month 
diftered, the variable was not changed. 

Where the student had at least 3 DOBs: 

1) If 2 of the DOBs matched and one differed, the 
DOB from the matched set was used. 

We then computed age as school year minus the date 
of birth year and then created overage variables as 
described. 

FINDINGS 

The percentage of overage DCPS students in grades 
6-9 has been slowly increasing, from an average of 40 
percent in 2005-06 to 43 percent in 2007-08. In 2007- 
08, about 1 in 4 6th graders was overage and more than 
half of 9th graders were overage (see Figure 1). 

In grades 6-8, a vast majority of overage students were 
1 year overage for grade and a very small proportion of 
students were 2 or more years overage (Figure 2). This 
pattern changes in 9th grade, where almost half of the 
overage students are 2 or more years overage. 

Using overage for grade as a warning signal, the data 
indicate that in 2007-08 more than half of DCPS 
students in grade 6-9 (more than 6,000 adolescents) are 
at risk of dropping out of high school before graduation 
(Figure 3). 



1 Rumberger, 1995. 

2 Roderick, 1 994; Shepard & Smith, 1 990. 

3 For a student to enroll in kindergarten, they must turn 5 years of age by December 3 1 st of that year. 

4 The age-cutoffs used to determine overage status were from Baltimore Public School’s work entitled Overage Students in the BCPSS. 
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FIGURE 1: PERCENTAGE OF STUDENTS OVERAGE BY GRADE LEVEL: 2005-06, 2006-07, 2007-08 




Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



FIGURE 2: PERCENT OF OVERAGE STUDENTS BY YEARS OVERAGE AND GRADE LEVEL, 2007-08^ 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



FIGURE 3: NUMBER OF OVERAGE STUDENTS BY YEARS OVERAGE AND GRADE LEVEL, 2007-08 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



5 The number of students on-age (and number of students with missing data) is as follows: 6th: 2190 (7); 7th: 1701 (17); 8th: 1950 (9); 9th: 2149 (18). 
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TOPIC IN FOCUS: AN EXAMINATION 
OF STUDENTS OVERAGE FOR GRADE 



AN INTRODUCTION TO THE SCHOOLS AND STUDENTS 
IN THE DISTRICT OF COLUMBIA 



In 2007-08, there were approximately 14,000 6th-9th graders enrolled in 1 16 public schools (see Tables A and 
B). Wards 5, 7, and 8 had the greatest number of 6-9th graders enrolled in DC public schools, as shown in Table C. 

Table A. Number DC Public School Students Table C. Student Enrollment in DC Public Schools 
Enrolled in Grades 6, 7, 8, or 9, 2007-08 by Grade and Ward, 2007-08 





N 


% 








Grade 






6th 


3128 


22% 


7th 


2751 


20% 


8th 


3145 


22% 


9th 


5054 


36% 


Total 


14078 


100% 



Table B. Number of DC Public Schoois Enrolling 
Students in Grades 6, 7, 8, or 9 by Grade Range, 
2007-08 
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STUDENT DEMOGRAPHICS 

Averaging across grades 6-9, males were about one and 
one-half times more likely to be overage than females 
(Figure 4). In 6th grade, twice as many male students 
were on-age than overage. By 9th grade, this had flipped; 
substantially more males were overage than on-age 
(33 and 19 percent respectively). By 9th grade, female 
students were divided evenly between on-age and 
overage. 



Overall, almost 70 percent of all DCPS students were 
poor (they qualified for the federal free or reduced-price 
lunch program) in 2007-08. Overage students were 
more likely to be poor than their on-age peers (Figure 5). 
Almost three-quarters of overage students in grades 6-9 
were poor; whereas, approximately two-thirds of on-age 
students were poor. These trends were consistent across 
grade levels and prior years. There was little difference 
between students' race and whether or not they were 
overage or on-age for grade. 



FIGURE 4: PERCENT OF FEMALES AND MALES ON-AGE AND OVERAGE BY GRADE LEVEL, 2007-08 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



FIGURE 5: PERCENT OF STUDENTS QUALIFYING AND NOT QUALIFYING FQR FREE OR REDUCED- 




Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



The Senior Urban Education Research Fellowship Series, Voiume I - Fall 2010 



29 



APPENDIX 



















TOPIC IN FOCUS: AN EXAMINATION 
OF STUDENTS OVERAGE FOR GRADE 



MATHEMATICS AND READING 
PROFICIENCY 

The most important data points in Table 1 are the 
percentages of DCPS students proficient in mathematics 
and reading— less than half of 6-8fh graders were 
proficienf in each contenf area in 2007-08T Averaged 
across grades 6-8, only about one-fifth of overage 
sfudenfs were proficienf in mathematics and one-quarter 
in reading. Approximately twice as many on-age students 
were proficient in mathematics and reading than overage 
students across all grade levels. 



The relationship between proficiency and overage status 
is clearly illustrated in Figures 6 and 7. In 22^ of the 
25 schools enrolling 8th graders, overage 8th graders 
exhibited lower proficiency than their on-age peers. The 
patterns indicated below are consistent with 6th and 7th 
grade proficiency rates. 



TABLE 1 : PERCENT (NUMBER) OF OVERAGE AND ON-AGE STUDENTS 
PROFICIENT IN READING AND MATH,° 2007-08° 



GRADE 


PROFICIENT IN MATH 


PROFICIENT IN READING 




OVERAGE 


ON-AGE 


ALL 


OVERAGE 


ON-AGE 


ALL 


6th 


22% (197) 


40% (628) 


37% (1095) 


29% (257) 


45% (101 3) 


43% (1270) 


7th 


19% (196) 


42% (702) 


36% (898) 


1 9% (200) 


44% (734) 


38% (934) 


8th 


20% (232) 


41% (803) 


36% (1035) 


23% (269) 


42% (827) 


38% (1096) 


Average 


20% (625) 


41% (2133) 


36% (3028) 


24% (726) 


44% (2574) 


40% (3300) 



Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



FIGURES 6 AND 7: 8TH GRADE STUDENTS’ PROFICIENCY IN MATHEMATICS AND READING: 
ON-AGE VS. OVERAGE, 2007-08 





Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



6 9th graders do not take the DC CAS. 

7 Schools that had higher percentages of overage students proficient than on-age students in math were Marshall Educational Center and Merritt Middle School and in reading: Browne Junior 
High School, Marshall Educational Center and Merritt Middle School. 

8 Eighth grade was chosen because while 9th grade had the highest percentage of overage students, and 8th grade was the tested grade with the highest percentage of students overage. 

9 The number of students not proficient in Math (and number of students with missing data) is as follows: 6th: 1 825 (208); 7th: 1 584 (269); 8th: 1 870 (240). In Reading, 6th: 1 650 (208); 7th: 
1548 (269); 8th: 1809 (240). 
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FIGURE 8: PERCENT OF 8TH GRADERS OVERAGE BY SCHOOL CHARACTERISTICS, 2007-08 




Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 



FIGURE 9: SCHOOL DEMOGRAPHICS, VIOLENT CRIME AND 8TH GRADE OVERAGE STATUS BY WARD, 2007-08 




Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University and Crime, 
Metropolitan Police Department 



SCHOOL CHARACTERISTICS 

Consistenf with the research,'® high-minority and high 
poverty schools (higher than average for DCPS) had 
greater concentrations of overage students than other 
DCPS schools. Schools with higher proportions of 
proficient students (again, higher than average for DCPS) 
had lower proportions of overage students. Figure 8 
illustrates this relationship for 8th graders; data for the 
other grades show similar patterns. 



WARD 

The District of Columbia is divided into 8 Wards, and as 
illustrated in Figure 9, there is substantial demographic 
variability among them. Ward 3, in particular, has much 
lower percentages of minority students, students 
receiving free/reduced-price lunch, and overage 
students than the other Wards. By contrast. Wards 1 , 5, 
7, and 8 are especially high on these measures. Violent 
crime follows the same trends. 



1 0 Alexander, Entwistle, & Dauber, 2003; Bali et al., 2005; Bianchi, 1 984; Corman, 2003. 
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TOPIC IN FOCUS: AN EXAMINATION 
OF STUDENTS OVERAGE FOR GRADE 



POVERTY AND CRIME IN THE DISTRICT OF COLUMBIA 

Overall, roughly one-fifth of all DC residents lived below poverty according to the latest Census. Forbes magazine 
named DC one of the top 1 0 most expensive cities to reside,' so the federally-set poverty level of $20,650^ for a 
family of four may under-represent DCs poverty. 



PERCENT OF PERSONS BELOW POVERTY LEVEL BY WARD, 2003 




DC has also been named one of the top 20 most dangerous cities for several years in a row.^ A majority of overage 
students attend school in the highest crime areas of DC. Research suggests that overage students are more likely 
to participate in committing crimes than their on-age peers.'' 




1 Forbes.com; http://www.forbes.eom/2009/1 0/07/most-expensive-cities-lifestyle-real-estate-america-top-ten_slide_2.html; downloaded March 1 2, 2010 

2 Federal Register, Vol. 72, No. 1 5, January 24, 2007, pp. 3147-3148 

3 QP Quarterly, 12, 13, 1 4th Annual Crime Statistics Report 

4 Juvenile Crime/Juvenile Justice (200 1 ). Commission on Behavioral and Social Sciences and Education. Behavioral and Social Sciences and Education. 
Washington, DC; National Academies Press. 
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Figure 9 indicates that more than half of the 8th graders 
in Wards 5 and 8 were overage and more than one-third 
of 8th graders were overage in Wards 1 , 4, 6, and 7. Ward 
3 had the lowest percentage of overage students (20 
percent). This pattern is similar for 6th and 7th grades 
as well. 

Map 1 illustrates the percentages of students overage 
by Ward, providing the location of schools and school 
concentration of overage students. 

In Ward 5, 1 3 percent of overage 8th graders were 
overage 2 or more years and in Ward 8, 10 percent 
of overage 8th graders were overage 2 or more years 
(Figure 1 0). Ward 3 had very few students overage by 2 
or more years. 



In 8th grade alone, nearly 1,200 students were overage 
for grade and at risk for dropping out, with close to a third 
of them attending school in Ward 8 (Figure 1 1). 

Figure 12 illustrates the relationship between 8th grade 
students’ overage status and mathematics and reading 
proficiency, disaggregated by Ward. Wards 5, 7, 8 had 
both the highest percentages of overage students and 
the lowest percentages of students proficient on the 
DC state assessment. Ward 3, on the other hand, had 
the lowest percentage of overage students and highest 
percentage of students proficient on the DC state 
assessment. 



MAP 1 : PERCENTAGE OF 8TH GRADE OVERAGE STUDENTS IN WASHINGTON, DC WARDS & SCHOOLS 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 

FIGURE 11: NUMBER OF OVERAGE 8TH GRADERS BY YEARS OVERAGE AND WARD, 2007-08 
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FIGURE 12: PERCENT OF 8TH GRADE STUDENTS PROFICIENT IN MATH, READING & OVERAGE FOR GRADE, 2007-08 
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Source: DCPS Dataset from DCPS Data Diagnostic, Center for Education Policy Research at Harvard University 
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CONCLUSIONS 

The mos1: significant and costly problem facing DCPS 
is the vast numbers of students dropping out of high 
school. An early warning signal that identifies students 
before they drop out will help DCPS leadership and 
school personnel target individual students and groups of 
students most in need of interventions and inform school 
improvement plans and decisions regarding intervention 
programs. Equally important, it will help district leaders 
plan for and make budget decisions regarding 
intervention programs, determine which schools to target 
for institution-level interventions and reforms, and inform 
planning for school-level and district-wide strategies. 

Data presented in this brief provide an overview of one 
type of early warning indicator, overage for grade, and 
illustrate the characteristics and concentration of these 
students in the District. This brief highlights both the 
enormity of the problem, but also pinpoints the schools 
and areas of the city where the problem is most prevalent. 
Using individual and real-time data, DCPS could be 
armed with the early warning data necessary to improve 
student graduation rates in the city. Though not sufficient 
to keep students in school, this early warning indicator 
could be the first step in building a system of student and 
school supports that will turn around failing schools and 
also redirect students toward college and career success. 
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